{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basics of MLP\n",
    "- Objective: create vanilla neural networks (i.e., Multilayer perceptrons) for simple regression/classification tasks with Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLP Structures\n",
    "- Each MLP model is consisted of one input layer, several hidden layers, and one output layer\n",
    "- Number of neurons in each layer is not limited\n",
    "<img src=\"http://cs231n.github.io/assets/nn1/neural_net.jpeg\" style=\"width: 300px\"/>\n",
    "<br>\n",
    "<center>**MLP with one hidden layer**</center>\n",
    "- Number of input neurons: 3\n",
    "- Number of hidden neurons: 4\n",
    "- Number of output neurons: 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://cs231n.github.io/assets/nn1/neural_net2.jpeg\" style=\"width: 500px\"/>\n",
    "<br>\n",
    "<center>**MLP with two hidden layers**</center>\n",
    "- Number of input neurons: 3\n",
    "- Number of hidden neurons: (4, 4)\n",
    "- Number of output neurons: 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLP for Regression tasks\n",
    "- When the target (**y**) is continuous (real)\n",
    "- For loss function and evaluation metric, mean squared error (MSE) is commonly used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import boston_housing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = boston_housing.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Description\n",
    "- Boston housing dataset has total 506 data instances (404 training & 102 test)\n",
    "- 13 attributes (features) to predict \"the median values of the houses at a location\"\n",
    "- Doc: https://keras.io/datasets/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(404, 13)\n",
      "(102, 13)\n",
      "(404,)\n",
      "(102,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Creating a model\n",
    "- Keras model object can be created with Sequential class\n",
    "- At the outset, the model is empty per se. It is completed by **'adding'** additional layers and compilation\n",
    "- Doc: https://keras.io/models/sequential/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1-1. Adding layers\n",
    "- Keras layers can be **added** to the model\n",
    "- Adding layers are like stacking lego blocks one by one\n",
    "- Doc: https://keras.io/layers/core/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers import Activation, Dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Keras model with two hidden layer with 10 neurons each \n",
    "model.add(Dense(10, input_shape = (13,)))    # Input layer => input_shape should be explicitly designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))                         # Hidden layer => only output dimension should be designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))                         # Hidden layer => only output dimension should be designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(1))                          # Output layer => output dimension = 1 since it is regression problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is equivalent to the above code block\n",
    "model.add(Dense(10, input_shape = (13,), activation = 'sigmoid'))\n",
    "model.add(Dense(10, activation = 'sigmoid'))\n",
    "model.add(Dense(10, activation = 'sigmoid'))\n",
    "model.add(Dense(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1-2. Model compile\n",
    "- Keras model should be \"compiled\" prior to training\n",
    "- Types of loss (function) and optimizer should be designated\n",
    "    - Doc (optimizers): https://keras.io/optimizers/\n",
    "    - Doc (losses): https://keras.io/losses/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sgd = optimizers.SGD(lr = 0.01)    # stochastic gradient descent optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer = sgd, loss = 'mean_squared_error', metrics = ['mse'])    # for regression problems, mean squared error (MSE) is often employed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_9 (Dense)              (None, 10)                140       \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 10)                110       \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 10)                110       \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 1)                 11        \n",
      "=================================================================\n",
      "Total params: 371\n",
      "Trainable params: 371\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Training\n",
    "- Training the model with training data provided"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.fit(X_train, y_train, batch_size = 50, epochs = 100, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Evaluation\n",
    "- Keras model can be evaluated with evaluate() function\n",
    "- Evaluation results are contained in a list\n",
    "    - Doc (metrics): https://keras.io/metrics/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      " 32/102 [========>.....................] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['loss', 'mean_squared_error']\n",
      "[81.900110581341906, 81.900110581341906]\n"
     ]
    }
   ],
   "source": [
    "print(model.metrics_names)     # list of metric names the model is employing\n",
    "print(results)                 # actual figure of metrics computed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss:  81.9001105813\n",
      "mse:  81.9001105813\n"
     ]
    }
   ],
   "source": [
    "print('loss: ', results[0])\n",
    "print('mse: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLP for classification tasks\n",
    "- When the target (**y**) is discrete (categorical)\n",
    "- For loss function, cross-entropy is used and for evaluation metric, accuracy is commonly used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "whole_data = load_breast_cancer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_data = whole_data.data\n",
    "y_data = whole_data.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, test_size = 0.3, random_state = 7) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Description\n",
    "- Breast cancer dataset has total 569 data instances (212 malign, 357 benign instances)\n",
    "- 30 attributes (features) to predict the binary class (M/B)\n",
    "- Doc: http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(398, 30)\n",
      "(171, 30)\n",
      "(398,)\n",
      "(171,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Creating a model\n",
    "- Same with regression model at the outset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1-1. Adding layers\n",
    "- Keras layers can be **added** to the model\n",
    "- Adding layers are like stacking lego blocks one by one\n",
    "- It should be noted that as this is a classification problem, sigmoid layer (softmax for multi-class problems) should be added\n",
    "- Doc: https://keras.io/layers/core/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Keras model with two hidden layer with 10 neurons each \n",
    "model.add(Dense(10, input_shape = (30,)))    # Input layer => input_shape should be explicitly designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))                         # Hidden layer => only output dimension should be designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))                         # Hidden layer => only output dimension should be designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(1))                          # Output layer => output dimension = 1 since it is regression problem\n",
    "model.add(Activation('sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is equivalent to the above code block\n",
    "model.add(Dense(10, input_shape = (13,), activation = 'sigmoid'))\n",
    "model.add(Dense(10, activation = 'sigmoid'))\n",
    "model.add(Dense(10, activation = 'sigmoid'))\n",
    "model.add(Dense(1, activation = 'sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1-2. Model compile\n",
    "- Keras model should be \"compiled\" prior to training\n",
    "- Types of loss (function) and optimizer should be designated\n",
    "    - Doc (optimizers): https://keras.io/optimizers/\n",
    "    - Doc (losses): https://keras.io/losses/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sgd = optimizers.SGD(lr = 0.01)    # stochastic gradient descent optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer = sgd, loss = 'binary_crossentropy', metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_13 (Dense)             (None, 10)                310       \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_14 (Dense)             (None, 10)                110       \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 10)                110       \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_16 (Dense)             (None, 1)                 11        \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 541\n",
      "Trainable params: 541\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Training\n",
    "- Training the model with training data provided"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.fit(X_train, y_train, batch_size = 50, epochs = 100, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Evaluation\n",
    "- Keras model can be evaluated with evaluate() function\n",
    "- Evaluation results are contained in a list\n",
    "    - Doc (metrics): https://keras.io/metrics/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      " 32/171 [====>.........................] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['loss', 'acc']\n",
      "[0.63870607063784235, 0.67836257240228481]\n"
     ]
    }
   ],
   "source": [
    "print(model.metrics_names)     # list of metric names the model is employing\n",
    "print(results)                 # actual figure of metrics computed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss:  0.638706070638\n",
      "accuracy:  0.678362572402\n"
     ]
    }
   ],
   "source": [
    "print('loss: ', results[0])\n",
    "print('accuracy: ', results[1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
